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Abstract 



This paper describes a place-based research demonstration program to promote and sus- 
tain employment among residents of selected public housing developments in six U.S. 
cities. Because all eligible residents of the participating public housing developments 
were free to take part in the program, it was not possible to study its impacts in a classical 
experiment, with random assignment of individual residents to the program or a control 
group. Instead, the impact analysis is based on a design that selected matched groups of 
two or three public housing developments in each participating city and randomly as- 
signed one to the program and the other(s) to a control group. In addition, an 11-year 
comparative interrupted time-series analysis is being used to strengthen the place-based 
random assignment design. Preliminary analyses of baseline data suggest that this two- 
pronged approach will provide credible estimates of program impacts. 
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THE POLICY PROBLEM 



The Jobs-Plus Community Revitalization Initiative for Public Housing Families is 
a place-based saturation-level employment demonstration program being tested in six 
cities across the United States. It was launched primarily to learn important lessons about 
addressing the problem of geographically concentrated joblessness and poverty. Al- 
though it focuses on public housing residents, the process through which the Jobs-Plus 
intervention was designed and implemented, how it is being evaluated, and certain fea- 
tures of the intervention itself point to a number of general lessons relevant to other 
community-based initiatives and institutional reforms. 

The basic Jobs-Plus model was designed jointly by the Manpower Demonstra- 
tion Research Corporation (MDRC) and the demonstration’s two core funding partners: 
The Rockefeller Foundation and the U.S. Department of Housing and Urban Develop- 
ment (HUD). 1 MDRC and other experts have provided extensive technical assistance to 
each participating city on the design and operation of its particular local approach. 
MDRC is also conducting a comprehensive evaluation of the program’s implementa- 
tion and effectiveness. 

The demonstration began in 1996 and will conclude in 2004. Participating sites 
have been operating the program since 1998. This paper summarizes the theory and pol- 
icy relevance of the project, the sites’ experiences to date in implementing the Jobs-Plus 
model, and the strategy being used to assess the intervention’s effectiveness in improving 
residents’ employment and quality-of-life outcomes and in helping to transform their 
public housing developments into better places to live. More detail on all of these issues 
can be found in the collection of evaluation reports and papers on Jobs-Plus that have 
been completed to date. (See Appendix A for a list.) 



Concentrated Joblessness and Poverty 

Paul Jargowsky opens his book, Poverty and Place, with the sobering observa- 
tion that: 

Every large city in the United States, whether economically vibrant or withering, 
has areas of extreme poverty, physical decay, and increasing abandonment. Most 
city residents will go to great lengths to avoid living, working, or even driving 
through these areas. 2 

William Julius Wilson, in his book, When Work Disappears, writes that: “for the 
first time in the twentieth century most adults in many inner-city ghetto neighborhoods 
are not working in a typical week,” adding that: “the current levels of joblessness in some 
neighborhoods are unprecedented.” 3 



'Other major funders are the U.S. Departments of Health and Human Service and Labor; the Joyce, James 
Irvine , Surdna, Northwest Area, Annie E. Casey, Stuart, and Washington Mutual Foundations, and BP. 

2 Jargowsky, 1 997, p. 1 . 

3 Wilson, 1996, p. xiii. 
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Even in good economic times, large cities across the U.S. include neighborhoods 
plagued by stubbornly high rates of joblessness and marginal employment, as William 
Dickens makes clear: “With national unemployment rates around 5 percent, it is not un- 
common to find neighborhoods where unemployment rates exceed 25 percent.” 4 

The concentration of joblessness and poverty has worsened in recent decades, 
with the number of high-poverty neighborhoods more than doubling between 1970 and 
1990, 5 a few years prior to commencement of the Jobs-Plus demonstration. During that 
period, members of racial and ethnic minority groups, especially African-Americans, 
were most immediately affected by this phenomenon. For example, in 1990, African- 
Americans made up about 13 percent of the U.S. population but accounted for more than 
half the population of high-poverty census tracts. 6 They also accounted for two-thirds of 
the population of urban census tracts where employment rates were lowest. 7 

Studies cite a host of external factors believed to have contributed to the spread of 
area-based poverty (although scholars continue to debate their relative importance). 
These include the decline of well-paying manufacturing jobs in the inner city; the con- 
centration of job growth in suburban areas not well linked to poorer communities by pub- 
lic transportation; the inadequate skills and preparation of inner-city residents for many 
of the new and better-paying service industry jobs; the flight of middle-class residents 
from center cities to the suburbs, leaving behind a poorer segment of the population; and 
the continuing legacy of racial discrimination, which restricts housing choices outside 
inner-city neighborhoods for minority group members. 8 

A growing body of literature suggests that living in high-poverty neighborhoods 
may contribute to poor social and economic outcomes for adults and their children (al- 
though the evidence is far from conclusive). 9 For example, Wilson suggests that where 
high rates of joblessness prevail, young people are cut off from role models and routines 
of life that can help socialize them for work. In that context, they may be more likely to 
resort to crime and other antisocial behaviors and to become teen parents. These behav- 
iors, in turn, can diminish prospects for completing school, acquiring skills, and moving 
into well-paying steady employment. 

Wilson and others also point to the likelihood that residents in poor areas are dis- 
proportionately isolated from social networks that can help them in the job market. For 
example, they often have fewer “connections” to people who can tell them about job 
openings (many of which go unadvertised) or who can serve as effective references by 



4 Dickens, 1999, p. 381. A recent study by HUD reports that 17 percent of central cities in larger met- 
ropolitan areas have unemployment rates 50 percent or more above the national unemployment rate, and 
that in 1995, 32 percent had poverty rates of 20 percent or more (HUD, 1999). 

5 Jargowsky, 1997, p. 30. Some researchers, including Jargowsky, define “high-poverty” neighbor- 
hoods or census tracts as those in which at least 40 percent of the population are poor. Others, such as Wil- 
son, 1996, and Turner, 1 998, set this threshold at 30 percent. 

6 Jargowsky, 1997, p. 39. 

7 Dickens, 1999, p. 382. 

8 See, for example, Wilson, 1996; Jargowsky, 1997; Dickens, 1999; Levy, 1998. 

9 See Brooks-Gunn, Duncan, and Aber, 1997 and Turner, 1998, for reviews of this literature. 
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writing a convincing letter of recommendation, “putting in a good word” with an em- 
ployer, or otherwise interceding on their behalf. 10 

Compounding these problems, inner-city residents may also be among the people 
most deeply affected by the recent sea change in U.S. federal policies for the poor, espe- 
cially time limits on welfare and other restrictions on access to safety-net benefits. These 
changes were enacted under the 1996 federal legislation that replaced the U.S. entitle- 
ment-based cash welfare system, Aid to Families with Dependent Children (AFDC), with 
its successor program, Temporary Assistance for Needy Families (TANF). 

Extent of the Problem in Public Housing 

The problems plaguing inner-city communities are particularly acute in many 
U.S. public housing developments (estates), which themselves rank among the most eco- 
nomically deprived neighborhoods in the country and are often part of larger neighbor- 
hoods with high rates of joblessness and poverty. In fact, around the time when Jobs-Plus 
was being launched, almost 54 percent of the United States’ 1.2 million units of public 
housing were located in high-poverty census tracts, and 68 percent were located in census 
tracts where 40 percent or more of working-age men had no regular employment. 1 1 

The population living in public housing has become substantially poorer in recent 
decades owing to the changing mission of public housing in the United States. Since its 
inception during the Great Depression, this strand of the nation’s social safety net has 
evolved from offering transitional shelter for unemployed workers to providing perma- 
nent housing for chronically nonemployed and impoverished people. Today, families 
with working members make up a minority of residents, especially in large inner-city hous- 
ing developments. Nationally, only about one-third of public housing families with children 
report wages as their primary source of income, whereas public assistance — including 
AFDC/TANF payments, state-provided General Assistance (GA), and Supplemental Secu- 
rity Income (SSI) — is the primary source of income for almost 50 percent of residents. 12 

In some cities, public housing residents appear to be among the hardest low- 
income persons to employ. 13 Many of them have a poor education and few job skills, 
meager work-relevant credentials, and an array of personal problems or situations that 
make it difficult for them to work. Furthermore, although empirical evidence is limited, it 
is widely believed that the mere circumstance of living in public housing directly im- 
pedes work — because of the stigma it casts in the eyes of many employers, the physical 



l0 Briggs, 1997 and 1998, draws the useful distinction between two dimensions of social capital: “so- 
cial leverage,” which is about access to information and influence that can help a person get ahead, and 
“social support,” which can help a person cope with difficult situations — for example, by providing emo- 
tional support or a small loan in an emergency. 

"Newman and Schnare, 1997. 

I2 U.S. Department of Housing and Urban Development, 1998. General Assistance is cash and/or in- 
kind support that some states and localities provide to eligible persons who do not qualify for federal cash 
assistance (such as single adults and childless couples). Supplemental Security Income is a federal program 
for low-income disabled adults. 

"Riccio and Orenstein, 2001 . 
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or social separation of residents from parts of the city or region where jobs are more 
abundant, and the absence of a social environment that promotes and rewards work. In 
addition, public housing rent rules, under which rent increases as earnings rise, have long 
been thought to discourage work. 

Increasing residents’ employment may be critical not only for helping them make 
progress toward self-sufficiency, but also for ensuring the future viability of public hous- 
ing as a source of decent, affordable shelter for low-income families. Some observers fear 
that, over the long term, time limits on welfare benefits that have been imposed by wel- 
fare reform policies in the U.S. will leave many residents with less income with which to 
pay their rent, and that federal budget problems may constrain federal operating subsidies 
to local public housing authorities (PHAs), making it harder for those authorities to fill 
the gap left by declining rent revenues. The resulting financial strain could foster a de- 
cline in the quality of housing services and living conditions, and perhaps even threaten 
PHAs’ very solvency . 14 

Jobs-Plus: A Response to the Problem 

Place-based economic self-sufficiency initiatives are an increasingly popular ap- 
proach for confronting geographically concentrated joblessness and poverty. 5 Jobs-Plus 
represents one such initiative, targeted toward a group of “low-work/high welfare” public 
housing developments. To date, such initiatives — inside or outside of public housing — 
have been modest in scale and scope. Hence, they have not moved large numbers of resi- 
dents into steady employment . 16 In addition, little has been learned about the effective- 
ness of these initiatives because the evaluation designs used to study them have not been 
able to produce convincing estimates of their impacts . 17 Jobs-Plus was launched to help 
fill this information gap. Its scale, scope and intensity were geared toward producing 
large impacts the employment, earnings and welfare receipt of public housing residents 
(with positive “spillover” effects on various quality-of-life outcomes for families and 
their housing development communities), 18 and its innovative evaluation design was 
geared toward producing valid and reliable estimates of those impacts . 19 



THEORY AND DESIGN OF THE INTERVENTION 

As the planners of Jobs-Plus set out to craft a new vision for combating high rates of 
joblessness and poverty in public housing, they sought to build upon lessons learned from 



14 See, for example, Naparstek, Dooley, and Smith, 1997. However, the U.S. General Accounting Of- 
fice (1998) reports that the effects of welfare reform on the need for additional operating subsidies for 
HUD’s housing subsidy programs are extremely difficult to forecast and existing empirical estimates vary 
widely. 

l5 Aspen Institute, 1997. 
l6 Riccio, 1999a and Newman, 1999. 
n Hollister.and Hill, 1995. 
l8 Riccio, 1999a. 
l9 Bloom, 1996; Riccio, 1998. 
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past carefully researched welfare-to-work and other employment programs for low-income 
populations. In addition, they tried to apply key principles from the growing number of 
comprehensive community initiatives being launched to improve the quality of life in poor 
urban neighborhoods. And like all community initiatives, Jobs-Plus hopes to achieve broad 
improvements in the quality of residents’ lives. It differs from the more typical approach, 
however, in that instead of attempting to achieve a variety of community change goals si- 
multaneously, it focuses on a single goal: improving employment outcomes. This is the 
driving force around which all program elements are organized. It is hypothesized (drawing 
on the work of Wilson and others) that by dramatically increasing employment, other im- 
provements in residents’ quality of life will follow, such as reductions in poverty and mate- 
rial hardship, crime, substance abuse, and social isolation; increased general satisfaction 
with living in the community; and improved outcomes for children. 

Drawing on Lessons from Past Employment Programs 

At the time Jobs-Plus was being designed, employment programs usually in- 
cluded several core features. Typical programs offered job search assistance (that is, in- 
struction and guidance in how to look for work, apply for jobs, and conduct oneself in job 
interviews); classroom-based education and training; and, to some extent, unpaid work 
experience or on-the-job training. Case management and subsidies for childcare and 
transportation to help recipients participate in programs were also common. In addition, 
most programs operating within the welfare system included participation mandates un- 
der which recipients faced possible reductions in their welfare grants if they failed to par- 
ticipate without “good cause.” 

Careful evaluations found that many such programs increased recipients’ earn- 
ings, reduced their welfare receipt, and more than paid for themselves. Mandatory wel- 
fare-to-work programs offering a mix of job search assistance, education, and training, 
with a clear and pervasive focus on relatively quick employment, were especially effec- 
tive. But while their gains were impressive, even the best-performing programs left sub- 
stantial numbers of recipients on the welfare rolls and did not greatly reduce the problem 
of high job turnover or the difficulty of moving from low-wage jobs to better-paying 
jobs . 20 

A number of subsequent initiatives adopted a broader vision of what it takes to 
help welfare recipients succeed in the labor market. Recognizing that leaving welfare for 
work at a low-paying job would not necessarily make recipients better off financially, 
most states in the U.S., as part of their TANF welfare reforms, have changed the way 
they calculate welfare grants in order to “make work pay.” Specifically, they allow more 
of a recipient’s earnings to be “disregarded” when the amount of the welfare grant is 
calculated. This means that more recipients are able to continue to receive welfare while 
working, and thus come out ahead financially by choosing to work. As its designers laid 
out Jobs-Plus, emerging results from a test in Minnesota of a program that combined such 
incentives with participation mandates and employment services looked promising, espe- 

20 See, for example, Bloom et at., 1993; Bloom, 1997; Hamilton et al., 1997; Scrivener et a)., 1998; 
Riccio, Friedlander, and Freedman, 1 994. 
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cially for a subgroup of urban welfare recipients who live in subsidized housing. At 18 
months, employment and earnings were higher and poverty rates lower for the program 
group in comparison to a randomly selected control group. 21 Other tests of interventions 
that incorporated financial incentives to “make work pay” were also underway in Canada 
and Milwaukee, Wisconsin. 22 These and other new strategies and their early evaluation 
results encouraged the designers of Jobs-Plus to incorporate a financial work incentives 
component into the Jobs-Plus model. But, as discussed below, the Jobs-Plus model goes 
even further. 

Drawing on Community-Building Principles 

Conceiving of Jobs-Plus as a place-based intervention with the goal not only of 
changing individuals but also of transforming the communities in which they live, the 
demonstration’s designers looked for further guidance to the efforts of a growing number 
of community change initiatives. The last several decades have seen the rise of numerous 
community efforts to revitalize poor urban neighborhoods and improve their residents’ 
quality of life. The earliest examples launched in the late 1980s by the Annie E. Casey 
Foundation (New Futures), Ford Foundation (Neighborhood and Family Initiative), and 
The Rockefeller Foundation (Community Planning and Action Program) helped inspire 
the emergence of an estimated 50 foundation-funded projects that have come to be 
known as “comprehensive community initiatives.” 23 “These initiatives,” one observer 
notes, “were different from past efforts in rejecting the notion that discrete ‘programs’ 
were the answer to urban poverty, in favor of a longer-term approach that builds commu- 
nity institutions, social networks, and residents’ self-reliance.” 4 Although their goals and 
tactics differ in the details, these initiatives tend to share a common set of “community- 
building” principles, which stress local control; collaborative decision making; resident 
empowerment; building on residents’ and communities’ existing physical, economic, and 
social assets; and strengthening the capacity of residents and local institutions to promote 
and sustain positive changes in their communities. 25 

The community-building focus of these projects drew inspiration from a growing 
body of research stressing the importance of “social capital” in the life of a community 
and for the well-being and economic advancement of the people living there. Unlike 



2l Miller et al., 1997; Miller, 1998. For longer-term results from this study, see Knox, Miller, and Gen- 
netian, 2000. 

22 In Wisconsin, the New Hope Program operated outside the existing public assistance system and was 
tested on a demonstration basis in two areas of Milwaukee. It included an earnings supplement, childcare 
subsidies, and affordable health insurance for eligible low-income people taking full-time jobs, and access 
to a temporary subsidized job for those who could not find full-time work in the unsubsidized labor mar- 
ket. An evaluation of the program found that it had positive effects on the employment, earnings, and in- 
come of people who were not working full time when they entered the program, and some positive effects 
on participants’ children, particularly boys (for example, improved behavior in school and higher educa- 
tional and occupational expectations; see Bos et al., 1999). For a description and final results of the Cana- 
dian experiment, see Michalopoulos et al., 2002. 

23 Aspen Institute, 1997; Walsh, 1997. 

24 Walsh, 1997, p. viii. 

25 Aspen Institute, 1997; Kingsley, McNeely, and Gibson, 1997; Walsh, 1997. 
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physical capital (such as factories, equipment, and commercial space) and human capital 
(such as job skills), social capital “inheres in the structure of relations between actors and 
among actors .” 26 Particularly important from a community perspective are aspects of so- 
cial capital such as residents’ engagement in formal and informal neighborhood organiza- 
tions (like churches, sports leagues, and parent-teacher associations), their personal 
friendship networks within and beyond the neighborhood, and relationships among larger 
community institutions (e.g., local businesses, schools, and the police). 

Robert Putnam popularized the concept of social capital with one study that drew 
a link between the level of civic engagement and the success of regional government and 
economic development in Italy, and another study documenting the decline of civic en- 
gagement in the United States . 27 Other community studies in the United States have 
found statistical relationships between the levels of certain aspects of social capital and 
neighborhood outcomes such as crime rates and neighborhood stability . 28 And a number 
of studies highlight the possible link between social networks in poorer communities (for 
example, where residents have fewer “connections” to people who can help them get 
jobs) and the lower employment rates and lower-paying jobs among people living there . 29 

Aware of the potentially powerful role that social networks might play in promot- 
ing — or thwarting — economic opportunities for residents of public housing, the Jobs-Plus 
designers added a third major component, which they called “community support for 
work.” Although they offered no blueprint specifying what forms this feature of Jobs- 
Plus should take, they did envision that, among other things, it would include involving 
the residents themselves in becoming sources of work promotion, encouragement, infor- 
mation, advice, and support to each other. In other words, Jobs-Plus would rely not just 
on professional caseworkers “doing things” to or for residents; it would also involve 
neighbors helping neighbors in ways that might improve their employment outcomes. 

The planners of Jobs-Plus also saw value in the emphasis that community- 
building initiatives place on enlisting and empowering community stakeholders in de- 
signing, funding, and operating the project. The principles of local collaboration, includ- 
ing resident involvement, call for key stakeholders to share the decisionmaking authority 
that controls the direction of the initiative, and for residents to play a central role, given 
their special knowledge of their own communities. But residents must work collabora- 
tively with institutional stakeholders (such as social service agencies, schools, commu- 
nity-based organizations, banks, businesses, hospitals, churches, the mayor’s office, and 
public housing authorities) that control resources and broader political influence affecting 
what can be accomplished. More generally, the joint efforts of a variety of institutions 
and systems, this view holds, can be much more effective than individual systems work- 



26 Coleman, 1988, p. S98. 

27 Putnam, 1995. 

28 See, for example, Sampson, Raudenbush, and Earls, 1997; Tempkin and Rohe, 1998. 

29 Dickens, 1999, p. 406, comments that “recent studies suggest that about half of all jobs and a larger 
fraction of good jobs are found through connections,” and that “persons displaced into unemployment rep- 
resent a double burden. They are no longer a source of information to the community about new jobs. And 
they are an additional burden to the network providing job referrals. . . . The net effect is that the escape 
from unemployment takes longer.” 



-7- 




13 



ing independently, and local funding may contribute to a sense of local ownership neces- 
sary to sustain such interventions over a long period of time, if they prove successful. 

Such collaboration, which has been a key feature of some past urban initiatives 
sponsored by the U.S. government (for example, the Community Action Program and 
Model Cities), appears to be enjoying a new prominence. It figures in the U.S. federal 
government’s HOPE VI program, which funds the replacement or reconstruction of dete- 
riorating public housing developments, and the federal Empowerment Zone/Enterprise 
Communities program, which funds economic revitalization efforts in distressed commu- 
nities. In each of these initiatives, residents and community groups are to be fully en- 
gaged with other community stakeholders in determining what gets done and how it gets 
done, and the approaches are to reflect more comprehensive visions for sustained com- 
munity development, not simply housing rehabilitation or economic development . 30 

The Jobs-Plus Intervention 

Based on these lessons from past research and principles of community building, 
Jobs-Plus was planned to be an unusually comprehensive and intensive community- 
focused employment intervention. 

Its three program components 

As indicated in the previous section, the program’s designers conceived of a 
broad, three-component intervention. One component focused on employment-related 
activities and support services such as instruction in job search skills, education and 
training, and assistance with childcare and transportation. Some of these services could 
be offered on site at the public housing developments, but the great diversity in residents’ 
job readiness and service needs also required access to broader networks of existing ser- 
vices. The second main program component involved financial incentives to “make work 
pay.” These comprised mainly new public housing rent rules that reduced the extent to 
which earnings gains would be offset by rent increases. The program’s third component, 
called community support for work, involved strengthening residents’ work-supporting 
social capital through means such as work-related information-sharing, peer support, and 
mutual aid among residents. 

Its saturation approach 

Jobs-Plus is also distinctive because of its attempt to implement all program com- 
ponents at saturation levels. That is, it was to be targeted toward all working-age resi- 
dents living in public housing developments selected to participate in the demonstration. 
Thus, at the very least, all such residents are to be exposed to new work-promoting “mes- 
sages” from program staff and neighbors. Furthermore, the families who participate can 
benefit from the new financial incentives and take advantage of a diverse array of ser- 
vices and supports. 



30 Kingsley et al., 1997; Naparstek, Dooley, and Smith, 1997. 



Providing the components of Jobs-Plus at saturation levels is fundamental to the 
program’s theory of change — the vision of how it is expected to produce unusually 
large impacts on employment and earnings. According to this theory, targeting the inter- 
vention toward the entire working-age population of a public housing development will 
produce a critical mass of employed residents (reaching a “tipping point”) 31 whose ex- 
periences will generate momentum for change across the development. As these van- 
guard workers grow in number, their visibility and role-model influence will be en- 
hanced. Their own success will signal to others the feasibility and benefits of working, 
elevate and strengthen social norms that encourage work, foster the growth of work- 
supporting social networks, and, ultimately, contribute to still more residents getting and 
keeping jobs. 

Its collaborative process 

From the outset, the demonstration’s planners decided not to attempt to make de- 
tailed design choices centrally. Instead, they chose to leave these decisions to local col- 
laboratives to be formed for this purpose. By requiring that each participating city tap a 
reservoir of local knowledge, technical expertise, and resources, the planners hoped that 
what emerged would stand a much greater chance of success than if any single local part- 
ner were to design and operate the program alone, or if it were to be designed centrally 
by the national demonstration team. 

Each local collaborative was expected to include a broad group of actors, but four 
partners were considered to be absolutely essential: the public housing authority, resident 
representatives, the welfare department, and the workforce development system (repre- 
sented by the agency operating since 1998 under the Workforce Investment Act, or 
WIA). Each of these partners could bring something special to the task of designing and 
implementing an effective Jobs-Plus program but was limited in what it could do alone. 
For example, the housing authorities had access to HUD resources and controlled many 
policies affecting housing developments and their tenants, but they needed the experience 
and resources of the welfare department and the workforce development agency in pro- 
viding employment and social services. At the same time, these agencies had little 
knowledge of the circumstances of public housing residents, who formed a sizable per- 
centage of their caseloads. Furthermore, resident representatives on the collaboratives 
could bring an in-depth awareness of their communities and service needs and could fos- 
ter community trust and “buy-in” for the program. Finally, other local organizations were 
expected to join as a source of services, expertise, and other resources that would help 
advance Jobs-Plus’s employment mission. 



31 Gladwell, 2000. 
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IMPLEMENTING THE INTERVENTION 



To implement Jobs-Plus required recruiting and choosing a group of eligible, ca- 
pable, and willing sites (cities), developing and maintaining a collaborative organization 
at each site, and building each of the three local program components. 

MDRC and the project’s core funders — the U.S. Department of Housing and 
Urban Development (HUD) and The Rockefeller Foundation — chose the sites from 
among a pool of interested and eligible cities. MDRC also deployed special “site repre- 
sentatives” and other experts to provide ongoing operations-related technical assistance 
to each collaborative to help it plan and implement the specific features of its Jobs-Plus 
program. 

Building local collaboratives and implementing new programs from the ground 
up are complicated, time-intensive enterprises, and the Jobs-Plus sites’ experiences were 
no exception. It took several years — much longer than had been hoped — for the program 
to evolve into a mature intervention that reflected the designers’ original vision. This 
long gestation period resulted in part from the slowness of the collaborative decisionmak- 
ing process; the challenges of meeting funding, staffing and space demands; and the chal- 
lenges of designing and integrating all the elements of the complex program model. 

Recruiting and Selecting Sites 

The planners of Jobs-Plus did not attempt to recruit cities and local housing au- 
thorities that, as a group, were nationally representative. Instead, they recruited a diverse 
set of sites where joblessness in public housing was a serious problem and where there 
appeared to be a good opportunity to build and test a large-scale, well-managed employ- 
ment initiative. 

Eligibility criteria 

Jobs-Plus sites were chosen through a national competition. Only large housing 
developments — defined as having at least 250 family-occupied units, not counting those 
occupied only by people 62 years old or older — could qualify. 32 In addition, no more 
than 30 percent of families living in these developments could have an employed mem- 
ber, and at least 40 percent had to be receiving AFDC. These criteria were meant to en- 
sure that Jobs-Plus would be tested in places where the need for an employment interven- 
tion was great and where the scale of the intervention could be substantial. 33 Across the 
continental United States, 442 housing developments managed by 53 local housing au- 
thorities met these criteria. 34 



32 A saturation strategy targeting all working-age residents would be considerably easier to implement 
in much smaller settings, but would be less valuable from a policy perspective. 

33 The sample-size needs of the demonstration’s evaluation design were another consideration. 

34 This estimate is based on MDRC calculations using 1993 data from HUD’s Information Services Di- 
vision of Public and Indian Housing. 



- 10 - 




16 



The quality of local PHA management was also important. Because Jobs-Plus 
was a complex and untried intervention, even the most effective housing authorities 
would be challenged by it. Thus, an effort was made to screen out PHAs that were having 
difficulty managing basic housing services. 

Furthermore, cities eligible for the demonstration had to be willing to adopt a 
collaborative strategy for designing and operating the intervention, and at least some of 
the key local partners had to have collaborated successfully in the past. The core role 
anticipated for the housing authority and the welfare and job training systems made 
their commitment essential. Cities also had to show a willingness to include residents 
as full partners, and existing resident organizations had to have a reasonable capacity 
to play that role. 

Finally, the local partners had to be willing and able to meet the demands of a 
rigorous research design. In particular, the housing authority had to have at least two— 
preferably three or more — developments that would qualify for Jobs-Plus, and (as dis- 
cussed later) MDRC had to be allowed to determine randomly which one of these 
would be selected to operate the program. One or two of the other developments would 
become part of a comparison group where research would be conducted but Jobs-Plus 
would not be operated. 

Candidate cities 

In June 1 996, an invitation to submit a statement of interest in the demonstration 
was sent to 50 of the 53 cities where, according to nationally available data, the public 
housing authority had the types of developments being sought. Attesting to the impor- 
tance that housing authorities and other city agencies ascribed to the project, positive re- 
sponses were received from 41 cities. 

After several rounds of information-gathering, in-depth site assessments, and in- 
ternal reviews, The Rockefeller Foundation, HUD, and MDRC chose 15 cities by August 
1996 to begin several months of preliminary program planning. During that period, these 
semifinalists received technical assistance from MDRC and other groups, in anticipation 
of submitting a formal application for the demonstration. 35 Of the 15 semi-finalists, six 
chose not to continue or were encouraged not to do so. 

Jobs-Plus developments 

In March 1997, seven cities — Baltimore, Maryland; Chattanooga, Tennessee; 
Cleveland, Ohio; Dayton, Ohio; Los Angeles, California; St. Paul, Minnesota; and Seat- 
tle, Washington — were selected to participate in the demonstration. At that point, the 



35 MDRC staff and consultants visited each of these 1 5 cities and also sponsored a cross-site confer- 
ence attended by key collaborative partners from each city, offering workshops and training sessions to 
help them think boldly and creatively about their initial program designs. After that conference, the sites 
were required to submit detailed, written applications in which they described their collaborates, gave 
evidence of local funding and resource commitments, and described their early vision of a Jobs-Plus pro- 
gram. 
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Jobs-Plus and comparison developments were selected randomly from the pool of candi- 
date developments for each city, and the main demonstration planning stage began. 

In 1999, due to a shift in local priorities, Cleveland left the demonstration by mu- 
tual agreement between its housing authority and the national Jobs-Plus team. In addi- 
tion, Seattle subsequently left the full demonstration because its housing authority re- 
ceived a federal HOPE VI grant to fund major renovations that will displace many resi- 
dents of its Jobs-Plus development. Seattle continues to run its Jobs-Plus program, but 
this program is now being evaluated separately from the program in other sites (although 
there continue to be many points of overlap). In sum, the full Jobs-Plus research demon- 
stration is operating in five of its seven original cities. In four of these cities the program 
is operating in one public housing development and in the fifth city, Los Angeles, it is 
operating in two housing developments. 

All of the Jobs-Plus developments comprise mainly low-rise units (in contrast to 
the popular image of public housing as agglomerations of high-rise towers). All but one 
of the Jobs-Plus developments is relatively large, however, each with more than 400 
households in residence. Several sites have a particularly good appearance, while others 
convey greater age and disrepair. And while some housing developments are close to 
commercial districts via public transportation, others are more isolated. 

Census data from 1990 indicate that the areas in which the Jobs-Plus develop- 
ments are located are similar to those featured in the literature on high-poverty communi- 
ties. As shown in Table 1, these are primarily census tracts populated by people of color. 
They are also tracts in which a high proportion of households are headed by single par- 
ents, many are living in poverty, and large numbers of adults do not have a high school 
diploma. Five of the seven developments are located in census tracts with poverty rates 
ranging from 49 to 74 percent, which is well above the 30 or 40 percent threshold com- 
monly used to designate “high-poverty” areas. 

Table 2 briefly describes the types of households that were living in the Jobs-Plus 
developments the sites were selected. As can be seen, they mirror the demographic com- 
position of the neighborhoods in which they are located. In addition, they comprise 
mainly female-headed households, with one adult member, plus several children. Perhaps 
most striking, however, is the very low percentages (15 percent to 25 percent) of house- 
holds receiving income from wages and the very high percentages (69 percent to 93 per- 
cent) receiving income from welfare (according to local PHA records). 
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Table 1 



Selected 1990 Characteristics of the Census Tracts 
in Which the Jobs-Plus Housing Developments Are Located 





Baltimore 


Chattanooga 


Dayton 


Los Angeles 


St. Paul 


Seattle 


Characteristic 


Gilmor 

Homes 


Harriet 

Tubman 

Homes 


DeSoto 

Bass 

Courts 


Imperial 

Courts 


William 

Mead 

Homes 


Mt. Airy 
Homes 


Rainier 

Vista 


Race/Ethnicity (%) 


Black, non-Hisp. 


98 


97 


97 


62 


34 


10 


19 


White, non-Hisp. 


0 


3 


1 


1 


18 


17 


25 


Hispanic 


1 


0 


1 


37 


42 


3 


4 


Asian 


0 


0 


2 


0 


5 


69 


50 


Single-parent 


households (%) 


43 


62 


52 


58 


37 


36 


21 


Adult high school 


graduates (%) 


53 


49 


58 


37 


59 


34 


71 


Household poverty 


rate (%) 


24 


58 


52 


59 


49 


74 


20 


Unemployment 


rate (%) 


18 


19 


26 


27 


10 


26 


6 



SOURCE: Tabulations for MDRC by the Center for Urban Research of the City University of New York, 
using the Atlas Select CD, a collection of 1990 census data. 

NOTES: The sample in each city includes residents of the census tract in which the Jobs-Plus development 
is located. Distributions may not total 100 percent because of rounding. Before rounding, the zero percent- 
ages ranged from 0.1 to 0.4 percent. Adult high school graduate rates are for persons age 25 or older. 
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Table 2 



Selected Characteristics of Household Heads and Households 
in the Jobs-Plus Housing Developments 
When the Sites Were Selected 





Baltimore 


Chattanooga 


Dayton 


Los Angeles 


St. Paul 


Seattle 


Characteristic 


Gilmor 

Homes 


Harriet 

Tubman 

Homes 


DeSoto 

Bass 

Courts 


Imperial 

Courts 


William 

Mead 

Homes 


Mt. Airy 
Homes 


Rainier 

Vista 


Household Heads 
















Race/Ethnicity (%) a 


Black, non-Hisp. 


99 


94 


98 


78 


6 


24 


42 


White, non-Hisp. 


0 


3 


1 


2 


1 


5 


12 


Hispanic 


0 


3 


0 


20 


80 


3 


0 


Asian 


0 


0 


0 


0 


13 


65 


43 


Female (%) 


79 


85 


88 


89 


60 


65 


74 


Elderly (%) b 


16 


12 


8 


8 


15 


8 


16 


Disabled (%) 


30 


27 


22 


16 


17 


27 


31 



Households 

Adults (%) 


One 


83 


89 


89 


71 


44 


46 


63 


Two or more 


17 


11 


11 


29 


56 


54 


37 


Children (%) 


None 


56 


35 


32 


23 


34 


10 


34 


One 


22 


22 


29 


25 


20 


13 


29 


Two 


14 


23 


22 


25 


21 


17 


18 


Three or more 


8 


20 


17 


27 


24 


59 


18 


Any income in past 
year from (%) 


Wages 


25 


20 


19 


15 


42 


16 


20 


AFDC 


46 


73 


56 


70 


46 


na d 


52 


Welfare c 


85 


90 


82 


93 


75 


87 


85 



SOURCES: Findings for the characteristics of household heads and the composition of households were 
obtained from MDRC calculations based on data from tenant rosters provided by housing authorities in 
October 1997. Findings for household income sources were obtained from housing authority data reported 
to MDRC in 1996 as part of their Jobs-Plus application. 

NOTES: Distributions may not total 1 00 percent because of rounding. 

distribution may not total 100 because other groups are not reported. 
b Persons 62 years of age or older. 

includes Aid to Families with Dependent Children (AFDC), state General Assistance (GA) pay- 
ments, and Supplemental Security Income (SSI), 
information not available. 
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Launching and Supporting the Local Collaboratives 

All sites included the four mandated Jobs-Plus partners in their collaboratives: the 
local public housing authority, the welfare department, the workforce development 
agency, and public housing residents. 36 They also included other local actors such as 
community foundations, nonprofit social service and employment and training providers, 
substance abuse treatment agencies, childcare agencies, and transportation agencies. Al- 
though selection of the lead partner was left to each local collaborative, all sites chose 
their housing authority. The degree to which the housing authority has been the “driving 
force” behind the initiative has varied across sites, however. 

In each site, some of these partners had worked together before, but rarely, if 
ever, had they all joined forces in pursuit of such an ambitious employment goal. Thus, 
how well the partnerships would function was uncertain. As it turned out, collaboration 
for Jobs-Plus has been a long and bumpy journey, with many challenges and setbacks. 
Early on, some partners left the collaboratives, seeing no concrete role for their organiza- 
tions. Others continued but expressed frustration at the slow pace of progress. Moreover, 
as a relatively small demonstration project, Jobs-Plus has had difficulty competing in 
some cities for the attention of senior agency officials who also have to contend with 
other local policy and administrative priorities. 

These problems (among others) contributed to the slow implementation of Jobs- 
Plus. Indeed, it took the collaboratives until the year 2000 or later to get elements of all 
three program components in place — several years after the sites were selected for the 
demonstration. 

Despite these difficulties, the collaboratives persevered and made important (if 
uneven) progress in jointly funding and shaping the Jobs-Plus program and in coordinat- 
ing services across agencies. The partners’ enduring commitment to this initiative can be 
traced largely to their converging interests in helping to increase employment among 
low-income people — many of whom live in public housing — particularly in the wake of 
welfare reform, which ended the entitlement to cash assistance. 

Collective decisionmaking 

The collaboratives initially structured themselves as formal governance bodies for 
making authoritative decisions over Jobs-Plus. In practice, the degree to which this oc- 
curred depended on the local housing authority’s willingness to share decisionmaking, 
the other partners’ desire to play a governing role, and the project director’s commitment 
to shared decisionmaking. Particularly during the program’s design phase, formal gov- 
ernance was important in giving “low-power” stakeholders like the residents and com- 
munity-based organizations an authoritative voice alongside large public agencies in de- 
veloping key aspects of the program. As the emphasis shifted from design to implementa- 
tion and ongoing development issues, strategic and operational decisions for Jobs-Plus 
increasingly shifted from the collaborative to the project director and staff in each site. 



36 This section draws heavily on Kato and Riccio, 2001, which provides a detailed analysis of the proc 
ess of collaboration in Jobs-Plus and offers guidance on this topic for other initiatives. 
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The extent to which the partners now play a formal governance role varies across the 
sites. But even where this role has been curtailed, the partners in all sites have continued 
to exert influence over Jobs-Plus in other ways, such as by providing ideas, expertise, and 
strategic advice through collaborative meetings that have come to be used as opportuni- 
ties for interagency networking and information-sharing; through staff contributed by 
partner agencies to Jobs-Plus; and through informal interactions with the project director. 

Involving residents as partners 

Residents were much more than “token” partners, but ensuring that they were in- 
volved in productive ways was a complex effort. Resident leaders had a particularly im- 
portant role in identifying the service needs of their community and proposing useful ser- 
vice approaches that were not obvious to professional staff operating under agency views 
of “what can be done.” Such leaders’ active support was critical in fostering residents’ 
trust and participation in the program. Involving residents effectively, however, chal- 
lenged the collaboratives to overcome considerable social and political obstacles. The 
barriers that residents faced to their broad, productive engagement varied across sites and 
included such factors as the exclusive professional culture of the agency representatives 
on the collaborative, entrenched resident leadership, adversarial relations with the hous- 
ing authority, and the residents’ need for greater technical expertise in order to advise an 
employment program. Thus, while it is feasible and critical to engage residents as col- 
laborative partners, making that happen requires the support of the institutional partners, 
certain skills and values on the part of the project director, and resident capacity-building 
efforts that develop specific leadership and management skills for performing well- 
defined roles in the program. 

Integrating services across providers 

The collaboratives in a number of sites took actions that improved the ways in 
which many different agencies worked together to deliver their services to residents of 
public housing. Although interagency service coordination for Jobs-Plus falls short of 
constituting a seamless, well-integrated network of services, the collaboratives helped to 
make changes in standard intake procedures and restructured the roles of frontline work- 
ers in key agencies to generate a more sensible division of labor among staff in jointly 
serving Jobs-Plus participants. The goal is to avoid placing duplicative — or contradic- 
tory-demands on residents. Joint staff training, better data-sharing, and building direct 
relationships among frontline staff across agencies also helped to coordinate services, 
construct sensible service plans, and monitor residents’ progress across services provided 
by a network of agencies. 



Meeting the Challenges of Funding, Staffing, and Space 

The long gestation period experienced by Jobs-Plus was due not only to the inevi- 
table complications involved in building new collaborative forms of decisionmaking and 
designing specific local strategies for a complex intervention, but also by the need to 
solve fundamental problems of infrastructure facing any new program. This includes 
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identifying resources to fund the program, defining and filling staff positions to operate 
it, and arranging for space in which to locate it. 

As previously mentioned, the designers of Jobs-Plus believed that the program 
would stand a better chance of being expanded and institutionalized after the demonstra- 
tion if it were funded during the demonstration with mainstream resources that local 
housing, welfare, and workforce agencies controlled. It was anticipated that these col- 
laborating agencies would be more likely to feel a sense of ownership and commitment to 
the program if they were investing their own resources in it. For the most part, these part- 
ners did make good on these funding commitments, and the bulk of resources to pay for 
Jobs-Plus services is coming from core public systems. At the same time, most of the re- 
sources represent “in-kind” rather than cash contributions. For example, a number of 
these agencies outstationed staff to work at the Jobs-Plus offices. The public housing au- 
thorities are also dedicating some of their existing staff to the program. Jobs-Plus also 
makes use of “slots” in existing programs operated by partner agencies. 

It has been more difficult for sites to obtain flexible funds needed to pay for new 
staff positions or special services that fall outside the usual offerings of participating 
agencies. To help address this problem, the demonstration’s funders provided each site 
with $200,000 for the period of the demonstration. In some cases, foundations and public 
agencies have provided additional small, flexible grants. For example, the U.S. Depart- 
ment of Labor has awarded small matching grants to the local workforce development 
agencies in each of the demonstration cities to help them help Jobs-Plus hire job develop- 
ers (staff who find jobs for clients). The sites also sought and won national, competitively 
awarded public grants, which they are dedicating fully or in part to Jobs-Plus. In addition, 
HUD is providing extra dollars to cover the costs of rent reductions instituted as part of 
the Jobs-Plus financial incentives component. 

The efforts to secure these funds partly explain why it took the sites two years or 
more to get a full complement of staff in place. The slow-moving hiring procedures of the 
housing authorities exacerbated these delays. After staff were brought on board, further 
time was needed to clarify their roles and program procedures. 

Acquiring locations for a Jobs-Plus office at each participating development also 
contributed to delays in implementing the program. In some sites, creating these spaces — 
which would serve as on-site “employment centers” that provided a convenient place for 
program staff and residents to meet, and for residents to meet among themselves — 
involved adapting and reconfiguring space in existing community centers. In others, 
apartment units were remodeled into office space. Negotiating for these spaces and the 
resources to make them suitable for Jobs-Plus required considerable staff attention. How- 
ever, it was universally accepted by the sites that, as a place-based initiative, Jobs-Plus 
must have a strong physical presence, which was essential to helping staff to feel like and 
be seen as a vital part of the community they were serving. 
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The Emerging Shape of the Program 

Designing and implementing a new package of job search, education, training, and 
support services for all working-age residents in a housing development would have been 
challenge enough. However, adding the two other components of Jobs-Plus — financial 
work incentives and community supports for work — increased the challenge several-fold. 

Employment-related activities and services 

The sites vary considerably in the specific kinds of employment-related services 
they offer and in the kinds of education and training providers they utilize. However, 
across the sites residents have tended to want and have been given assistance by Jobs- 
Plus to try to find work as quickly as possible. Individualized job search guidance is a 
core program feature at all sites. Some sites have also provided or referred residents to 
group-based job clubs, which teach job-hunting and interviewing techniques. Instruction 
(directly or through referral to other organizations) is also offered in “soft” employment 
skills, such as understanding employers’ expectations and appropriate workplace behav- 
ior. In addition, sites are trying to make job listings easily available to residents, such as 
by providing access to computers in Jobs-Plus resource centers to help residents search 
for jobs posted on the Internet. 

To supplement residents’ individual job-search efforts, local programs offer assis- 
tance from professional job developers that identify employers who are looking for work- 
ers in positions for which Jobs-Plus participants would be suitable. They also try to gen- 
erate special employment opportunities by encouraging employers to recruit from the 
Jobs-Plus program when future vacancies occur. 

Although Jobs-Plus encourages quick employment, it also helps residents partici- 
pate in education and training activities — in ways that do not keep them out of the labor 
market for long. Higher priority is thus given to short-term training, and to combining 
work with education or training concurrently (for example, working during the day and 
attending classes at night or on weekends). Because this is easier to do if training is lo- 
cated at the Jobs-Plus development, some programs are offering part-time, on-site basic 
education classes outside normal business hours. 

In addition, Jobs-Plus offers a broad range of support services to help residents par- 
ticipate in program activities as well as find and keep jobs. Most common among these ser- 
vices are childcare, transportation, substance abuse treatment, and domestic abuse assis- 
tance. 



Financial incentives to make work pay 

To encourage residents to take jobs and remain employed, Jobs-Plus includes new 
rent policies that limit how much residents’ rent will increase when their incomes rise. 
Under traditional rent policies, residents must pay 30 percent of their household’s count- 
able income in rent, up to a maximum amount tied to the cost of operating public hous- 
ing. Under Jobs-Plus, residents pay less of their overall income in rent. Depending on 
how many people are induced by Jobs-Plus to work, it is possible for the housing author- 
ity to lose revenues as a result of the program. Recognizing that few housing authorities 
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would be willing to take this risk, HUD agreed to hold them harmless for any extra costs 
that resulted from approved new Jobs-Plus rent policies. 37 

Current Jobs-Plus incentive packages center primarily on two main strategies, 
with different sites taking different approaches: (1) flat rents, which specify a fixed rental 
payment regardless of earnings, and (2) reductions in the percentage of income to be paid 
in rent. 38 Residents must participate in other Jobs-Plus activities in order to qualify for 

IQ 

these rent benefits. 

Another important feature of the Jobs-Plus work incentives approach is a con- 
certed effort by sites to educate their residents about the other financial programs for low- 
income working families and individuals that exist under current law (such as earnings dis- 
regards available under TANF, assistance with the cost of childcare and Medicaid that wel- 
fare recipients can continue to receive after leaving welfare, childcare disregards under pub- 
lic housing rent rules, and the Earned Income Tax Credit, or EITC). Each site has devel- 
oped a strategy to help residents take advantage of these incentives in addition to those 
available through Jobs-Plus. 

Community support for work 

As noted earlier, the third main component of Jobs-Plus is an explicit attempt to 
increase “community support for work.” Because this concept is imprecise and means 
different things to different people, it was no surprise that sites chose to implement the 
other two program components first. However, as those components began to take shape, 
it became possible to link community support for work to them. Two basic approaches 
were used to do so: (I) strengthening residents’ social networks, and (2) changing institu- 
tional conditions that were creating barriers to employment. 

Residents’ social networks can become sources of information about work oppor- 
tunities, sources of help in understanding work incentives, and sources of encouragement 
to work and assistance in solving practical, personal, or interpersonal problems. Toward 
this end, the sites have focused on the establishment of a network of resident “community 
coaches.” These residents, who function as outreach workers, engage in a process of 
“neighbor-to-neighbor” information-sharing and support that is focused explicitly on 
work. This includes disseminating throughout the development information about con- 
crete job openings and about opportunities available through the Jobs-Plus program. As a 
way to focus the community coaches’ efforts specifically on employment-related activi- 
ties, sites have been encouraged to adopt a new job-related outreach “campaign” each 



37 Issues arose however, between HUD and the U.S. Congressional committee that oversees its depart- 
mental budget over how to cover these costs (though not over the “hold-harmless” concept), leading to 
several months of negotiations and, consequently, delays in sites’ ability to implement their incentive 
plans. The funding problem was eventually solved, and a final agreement between Congress and HUD was 
reached in May 1 999. 

38 Miller and Riccio, 2002, provide a detailed description of the incentive package offered by each site. 

39 Public housing rent reform continues to be of great interest in the U.S. For example, the federal 
Quality Housing and Work Responsibility Act of 1 998 requires local housing authorities to introduce cer- 
tain new rent policies that are more favorable for working families. Hence, the Jobs-Plus experience with 
rent incentives is directly relevant to housing policy more broadly. 
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month — for example, on promoting rent incentives, on-site employer recruitments, or in- 
formation on the E1TC. These campaigns have now become the centerpiece of commu- 
nity outreach activities. This work supplements other efforts at the developments, includ- 
ing community-wide events such as job fairs, health fairs, and holiday and back-to-school 
events that all sites continue to sponsor as a way of fostering a stronger sense of commu- 
nity among residents. In one site, the community coaches have also spearheaded efforts 
to organize basic education classes at the development, and they recruit other residents to 
participate in those classes and provide childcare for those who need it while the classes 
are in session. 

In all sites, the combined efforts of Jobs-Plus staff and resident outreach workers 
have helped to create an environment within the participating public housing communi- 
ties in which work is strongly promoted and “messages,” at least, about employment- 
related assistance and opportunities abound. Qualitative evidence from the evaluation’s 
ongoing implementation research suggests that knowledge about Jobs-Plus as a place to 
go to get help with one’s employment needs is widespread among the tenant population 
in each development — an important achievement given the neighborhood-saturation fo- 
cus of the intervention. 



DESIGN AND IMPLEMENTATION OF THE IMPACT 

EVALUATION 

Given the complex, multifaceted nature of the Jobs-Plus intervention and its im- 
plementation, there is no single simple evaluation strategy capable of measuring its im- 
pacts. Thus, we are: (1) estimating program impacts from two different perspectives — 
that of individual public housing residents and that of the developments in which they 
live (people and place); (2) basing these estimates on an evaluation strategy that com- 
bines two complementary approaches — random assignment of housing developments 
plus a comparative interrupted time-series analysis; and (3) relying on two main sources 
of outcome data — surveys of housing residents’ attitudes, perceptions and experiences 
plus government administrative records on their employment, earnings and welfare re- 
ceipt. 

Measuring Impacts from Two Perspectives: People and Place 

A central feature of the Jobs-Plus evaluation design is its focus on impacts from 
two different perspectives: (1) with respect to specific individual public housing residents 
(people), and (2) with respect to specific public housing developments (place). 

The individual perspective relates to a particular group of persons who were liv- 
ing in Jobs-Plus developments at a specific point in time. Thus, it focuses on a single 
resident cohort. From this perspective, the Jobs-Plus impact analysis will address the 
question: “How did the demonstration program affect the future experiences of its target 
individuals, whether or not they moved away?” 
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The housing development perspective relates to groups of different persons who 
were living in Jobs-Plus developments at different points in time. Thus, it focuses on a 
series of consecutive, partly overlapping resident cohorts. From this perspective, the 
Jobs-Plus impact analysis will address the question: “How did the demonstration pro- 
gram affect conditions in its target developments, given that different people were living 
there at different times?” 

The distinction between these two perspectives is key to any evaluation of a 
place-based initiative because sample members can move into and out of its target area. 
For example, students can move into and out of schools that are implementing whole- 
school reforms; families can move into and out of communities that are implementing 
health education programs; and employees can move into and out of firms that are im- 
plementing worker retention programs. 40 In these ways, mobility drives a conceptual and 
operational wedge between people and place. For example, during 1998, when Jobs-Plus 
was being launched, between 13 percent and 36 percent of the working-age, nondisabled 
household heads moved out of its program developments. 41 

Randomly Assigning Housing Developments: An Approach 
to Measuring Average Impacts Across Sites 

In the field of employment and training research, random assignment experiments 
are now widely regarded as the best way to estimate program impacts. In the words of 
one prominent researcher, this approach is “a bit like the nectar of the gods: once you’ve 
had a taste of the pure stuff it is hard to settle for the flawed alternatives.” 42 However, 
Jobs-Plus is not a program to which individuals or households can be assigned randomly. 
Instead, it is a place-based initiative to which all eligible residents of participating public 
housing developments may be exposed and in which all eligible residents can take part if 
they desire. 

It was possible, however, to select an approximately matched group of two or 
three candidate housing developments for each Jobs-Plus site, and then randomly choose 
(using computer-generated random numbers) the development that would launch the in- 
tervention. The remaining developments could thus serve as a comparison group. As 
noted earlier, the ability for MDRC to make this choice randomly was an eligibility re- 
quirement for each site, and all 15 cities that became semifinalists accepted this require- 
ment. Although explaining the need for the requirement took considerable time and effort, 
it was generally recognized that Jobs-Plus represented a scarce resource whose allocation 
by lottery was ethical and fair. In addition, anecdotal evidence suggests that at least some 
local residents felt that random selection of the participating housing development would be 
fairer than selection by the public housing authority — which was not always trusted. 



40 See Bloom, forthcoming, and Bloom, Bos and Lee, 1 999, with respect to evaluating whole-school re- 
forms. See Murray et. al., 1994, with respect to evaluating a health education initiative. Lastly, note that 
MDRC is currently evaluating an employer-based worker retention program in Cleveland, Ohio, but writ- 
ten documentation on the project is not yet available. 

4l Verma, 2002a, Figure 1. 

42 Hollister and Hill, 1995, p. 134. 
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Table 3 lists the Jobs-Plus and comparison developments for each site. As can be 
seen, four sites have a randomly selected Jobs-Plus development plus two comparison 
developments; one site has a randomly selected Jobs-Plus development plus one com- 
parison development; and one site (Los Angeles) has two Jobs-Plus developments, one of 
which has no comparison development (the findings for this Jobs-Plus development will 
not be combined with those for the others). 

It thus will be possible to estimate impacts for each site by comparing outcomes 
for its Jobs-Plus and pooled comparison groups. In addition, it will be possible to pool 
these impact estimates across sites by taking their average. For both types of impact es- 
timates, random assignment of Jobs-Plus and comparison developments protects against 
the possibility of “stacking the deck” by consciously or inadvertently choosing Jobs-Plus 
developments that are more likely or less likely than others to improve their future out- 
comes without the intervention. Other than this protection (which is not trivial), site- 
specific impact estimates are nonexpert mental or quasiexperimental, and thus subject to 
the methodological threats inherent in such designs. 



Table 3 

Jobs-Plus and Comparison Housing Developments, by Site 



Site 


Jobs-Plus 

Development(s) 


Comparison 

Development(s) 


Baltimore 


Gilmor Homes 


Perkins Courts 
Somerset Courts 


Chattanooga 


Harriet Tubman 


College Hill 
Emma Wheeler 


Dayton 


DeSoto Bass 


Arlington Courts 
Parkside Courts 


St. Paul 


Mount Airy 


McDonough 

Roosevelt 


Seattle 


Rainier Vista 


Yesler Courts 


Los Angeles 


William Mead 


Dana Strand 




Imperial Courts 


None 



NOTE: Cleveland, which is no longer in the demonstration, had one randomly selected Jobs-Plus devel- 
opment (Woodhill Homes) plus two comparison developments (Garden Valley and Riverside Park). Impe- 
rial Courts, in Los Angeles, was chosen to implement a Jobs-Plus program, but did not have a randomly 
assigned comparison development. 



Random assignment of housing developments can produce unbiased (internally 
valid) estimates of average impacts pooled across the demonstration sites. This means 
that the long-run expected value of the impact estimator (a theoretical property) equals 
the “true” average value of the impacts. At the same time, this type of random assignment 
of groups (often referred to as “cluster” random assignment) produces impact estimates 
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that have less statistical power — and, thus, greater uncertainty — than those based on in- 
dividual random assignment of the same number of persons. 

Although these theoretical properties are well documented, 43 just how much sta- 
tistical power is lost when moving from individual-level to group-level random assign- 
ment is an empirical question. This loss of power depends on three factors: (1) the degree 
to which individual outcomes vary within versus across groups (measured by their intra- 
class correlation); (2) the number of groups being randomly assigned (fewer groups pro- 
duce less power); and (3) the extent to which the variance of the outcome within and 
across groups is reduced by statistical controls for preexisting individual-level or group- 
level characteristics. Bloom, Bos and Lee provide a detailed empirical analysis of these 
conditions for the random assignment of schools and find that: 

If a good measure of past individual or school performance is available, it might 
be possible to detect a 3- to 6-percentile improvement in average student per- 
formance with cluster assignment of 40 schools . . . (half to the program group 
and half to the control group) . . . and 60 students per school (2,400 students over- 
all). This implies an effect size of roughly 0.10 to 0.20, which by most existing 
standards suggests adequate statistical power. 44 

In addition, they find that school-level measures of past student performance are almost 
as effective as individual-level measures with respect to statistical power. 

The Jobs-Plus impact analysis will have six program developments and ten com- 
parison developments (including Seattle for parts of the follow-up period). This small 
number suggests that impact estimates based on simple or regression-adjusted outcome 
contrasts for the Jobs-Plus and comparison groups may have limited statistical power. On 
the other hand, there are many hundreds of sample members for each development and 
hence, the overall sample of individuals is quite large. This suggests potentially higher 
statistical power. Furthermore, baseline and follow-up data are available for certain out- 
come measures, which will make it possible to control statistically for preexisting varia- 
tion within and between developments. This suggests potentially greater statistical power. 
One final factor, which also suggests greater statistical power, is that developments were 
approximately matched and blocked by city before they were randomly assigned. This 
eliminates city-level outcome differences from subsequent impact estimates and thereby 
increases statistical power. 

The degree to which future outcomes will vary within and across developments 
(their intraclass correlation) is not yet known. As noted above, this factor has a major in- 
fluence on the degree to which randomly assigning groups instead of individuals reduces 
the statistical power of program impact estimates. Even more importantly, given the find- 
ing of Bloom, Bos and Lee, is the extent to which individual-level and development-level 
baseline characteristics can reduce the unexplained (random) variation in outcomes 
within and across housing developments. In other words, what really counts is the “con- 
ditional” intra-class correlation after statistical controls for baseline covariates have been 



43 Raudenbush, 1997, presents a clear exposition of these properties. 
““Bloom, Bos and Lee, 1999, p. 464. 
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applied. Thus, it is not yet clear just how much statistical power will exist for pooled es- 
timates of Jobs-Plus impacts based solely on the random assignment of developments. 

Comparative Interrupted Time-Series Analysis: An Approach 
to Measuring Site-Specific Impacts 

To produce rigorous site-level estimates of Jobs-Plus impacts (and thereby further 
enhance corresponding pooled estimates), we are implementing a comparative inter- 
rupted time-series analysis. Interrupted time-series analysis is a quasiexperimental design 
that has been used successfully to evaluate programs in many fields. 45 It is based on mul- 
tiple baseline observations of an outcome before a program is launched plus one or more 
follow-up observations after the program is launched. Our application of the approach 
will proceed in two steps: (1) we will measure the extent to which the follow-up outcome 
measure (for example, the employment rate) in a Jobs-Plus development deviates from its 
baseline trend; and ( 2 ) we will compare the observed deviation from trend in the Jobs- 
Plus development with its counterpart for the comparison group. The first step in this 
process addresses the question: “To what extent was there an improvement in the experi- 
ences of residents at the program development?” The second step addresses the question: 
“To what extent did Jobs-Plus improve these experiences?” 

Estimating the Jobs-Plus deviation from trend 

The simplest interrupted time-series analysis involves a single Jobs-Plus devel- 
opment with multiple periods of data before and after the intervention was launched. This 
analysis can be applied to numerous outcome measures such as employment rates, mean 
earnings and welfare receipt rates. Figure 1 illustrates how to use the analysis to estimate 
shifts in the percentage of residents employed. 

With at least six years of baseline data on this measure, it is possible to fit a pre- 
Jobs-Plus baseline trend line. A linear trend may be adequate for many outcomes, but a 
curvilinear trend can be fit, under some conditions, if the curvature of the baseline pattern 
is pronounced. Extrapolation of the baseline trend provides the best available estimate of 
what the outcome would have been without any major economic or policy changes. 46 

The deviation from the baseline trend in the first year after Jobs-Plus begins (line 
Di in the figure) provides an estimate of the “shift” in the outcome for that that year. De- 
viations from trend in subsequent years (lines D2 through D5) provide corresponding es- 
timates of the shifts for these years. Estimating the shift for each follow-up year provides 
an easy way to describe the pattern of unpredicted change over time (whether it is con- 



45 Shadish, Cook and Campbell, 2002, provide a comprehensive review of the interrupted time-series 
literature. Campbell and Stanley, 1966, and Cook and Campbell, 1979, are perhaps the most widely read 
sources about the approach. Bloom, 2002, forthcoming, describes how to use it to measure the impacts of 
whole-school reforms on student performance. 

46 This implies that the best predictor of future behavior is long-term past behavior, which is the case 
for many outcomes. 
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stant, it decays, or it grows). One can then average these estimates to summarize them for 
the follow-up period . 47 

For example, the hypothetical results in the bottom panel of Figure 1 indicate that 
there was a 3-point increase in the percentage of residents who were employed in the first 
year after Jobs-Plus began (i.e., year seven of the data collection period), a 7-point in- 
crease in the second year, and so on, ending with a 17-point increase in the fifth year af- 
ter the program began. Hence, the outcome was about 1 1 percentage points higher than 
predicted, on average, during the following period. 

The following regression model can be used to estimate the shifts in the figure: 

Y« = GB-HE 0 t + l 8 iP, + e, (1) 



Where: 

Y t = the value of the outcome variable in year t, 

P t = one if year t is after Jobs-Plus began and zero otherwise, 

t = the year, 

e, = a random error term, 

150 = the slope of the baseline trend, 

151 = the deviation from trend after Jobs-Plus began, 
aa = the intercept of the baseline trend. 

If only years zero through seven are included in the analysis, the coefficient, II! i, 
equals the deviation from trend in year seven (line Di, in the figure), and the t-statistic for 
this coefficient provides a test of its statistical significance. To include all five follow-up 
years in the analysis and allow each to have a separate deviation from trend, one can re- 
place P with a separate dummy variable for each year. The coefficient for each dummy 
variable equals the deviation from trend for the year that it represents (lines Di through 
D 5 ), and the t-statistic for each coefficient provides a test of its statistical significance. 



47 If there are at least three follow-up observations, it is possible, in theory, to estimate the impact of the 
program on the intercept and slope of the original trend-line. We do not take this approach, however, be- 
cause it does not focus on the actual annual impacts and, hence, is more difficult to interpret. 
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Figure 1 

Illustration of an Interrupted Time-Series Analysis 
for a Single Jobs-Plus Development 




YEARS 



Results During the Follow-up Period 



Year 


Predicted 

Outcome 

(percent) 


Actual 

Outcome 

(percent) 


Deviation from 
Trend 

(percentage- points) 


7 


51 


54 


+ 3 


8 


53 


60 


+ 7 


9 


56 


68 


+ 12 


10 


55 


70 


+ 15 


11 


57 


74 


+ 17 



For an interrupted time-series analysis to be most effective there must be a stable 
baseline trend and a pronounced deviation from this trend. The more stable the baseline 
trend is (the less the points vary around the trend-line), the more confidence one can 
place in the forecast or extrapolation for the follow-up period. The larger and more 
abrupt the deviation from trend is, the easier it will be to identify. 
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Comparing the Jobs-Plus and comparison group deviations from trend 

A logical extension of the preceding approach is to conduct a separate interrupted 
time-series analysis for the comparison group where Jobs-Plus was not implemented 
(pooling the samples of residents for the two comparison developments for each site 
where there are two). Figure 2 illustrates how this time-series analysis can be used to 
produce estimates of the impacts of Jobs-Plus (the shift in outcomes that it caused). The 
approach is applicable regardless of how the comparison development was chosen (with 
or without random assignment and/or matching). 

The top panel of Figure 2 repeats the time-series analysis for the hypothetical 
Jobs-Plus development in Figure 1. The bottom panel presents findings during the 
same period for its comparison group. The interrupted time-series analysis for the 
comparison group yields deviations from trend in years 7, 8 and 9 equal to E|, E2 and 
E3, respectively. If the comparison group and the Jobs-Plus development were chosen 
from the same local environment, then the comparison group’s deviation from its trend 
provides an estimate of what the deviation from trend would have been for the Jobs- 
Plus development without Jobs-Plus (the counterfactual for our impact estimates). 
Hence, D t - E t provides an estimate of the impact of Jobs-Plus in year t. The variance 
of this difference equals the sum of the variances of D t and E t . 48 Thus, one can readily 
test the statistical significance of the difference. 

The slow implementation of Jobs-Plus described in earlier sections of this pa- 
per — particularly the fact that not all features of the program model were in place until 
several years after site selection — means that an immediate deviation from the baseline 
employment trend in the Jobs-Plus developments is unlikely. It is thus fortunate that the 
follow-up period for the impact evaluation will extend a full five years after program op- 
erations began. However, impacts are likely to emerge gradually over time, rather than 
abruptly. This will make it difficult for the interrupted time-series analysis to clearly 
identify them because they may not reflect pronounced deviations from the baseline 
trend. Nevertheless, if these impacts are experienced at all or most of the Jobs-Plus sites, 
pooling site-specific findings will improve our chances of accurately detecting them. 

Pooling site-specific findings 

Once site-specific impact estimates have been obtained using the preceding ap- 
proach, we will pool them across sites by taking their mean. This will produce estimates 
of the average Jobs-Plus impact for the study sites. These core evaluation findings will 
have the combined methodological protection of the random assignment of housing de- 
velopments (which produces unbiased impact estimates), and the comparative interrupted 
time-series analysis (which provides further protection against bias and most likely also 
increases statistical power). 



48 This simple formulation is possible because the two developments comprise independent samples. 
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Figure 2 

Illustration of an Interrupted Time-Series Analysis 
for a Jobs-Plus Development and a Comparison Development 
During the Jobs-Plus Baseline Period 




YEARS 




YEARS 
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Data Sources 



Given the focus of Jobs-Plus on resident employment, the primary outcomes for 
its impact analysis will be employment, earnings, and welfare receipt. For each of these 
outcomes, a comparative interrupted time-series analysis will be used to estimate pro- 
gram impacts. For this purpose, quarterly data on employment and earnings will be ob- 
tained from the administrative records of state Unemployment Insurance (UI) agencies 
for a baseline period of roughly six years before Jobs-Plus was launched and a follow-up 
period of roughly five years thereafter. These data, which are reported quarterly by em- 
ployers in all states to their state unemployment insurance agency, cover' well over 90 
percent of all jobs in the formal labor market and have been found in past research to 
provide adequate information for measuring program impacts. 49 In addition, monthly data 
on the receipt of AFDC/TANF payments and food stamps during the same period will be 
obtained from the administrative records of state and local welfare agencies. This type of 
data has been used for many past evaluations of welfare-to-work and employment pro- 
grams and is generally thought to be accurate and complete. 

Local PH A records will be used to obtain a limited set of background characteris- 
tics on residents, such as age, gender, race/ethnicity, how long they have lived in their 
current development, whether they move from it subsequently, and whether their house- 
hold is receiving welfare. This information will be used to construct selected subgroups 
for the impact analysis. We refer to information obtained from UI wage records, welfare 
payments records, and local PHA records as administrative data. 

In addition to this information, a baseline survey has been conducted and a fol- 
low-up survey will be conducted in the Jobs-Plus and comparison developments. Infor- 
mation from the surveys includes measures of: (1) community life, (2) outcomes and ac- 
tivities for children, (3) residents’ employment and the characteristics of their jobs, (4) 
family income and material well-being, (5) individual physical and mental health, (6) in- 
dividual background characteristics and, (7) individual participation in education, train- 
ing and employment-related activities. 

The baseline survey was administered in the spring and summer of 1998 to a rep- 
resentative sample of household heads living in the Jobs-Plus and comparison develop- 
ments. The follow-up survey will be administered in 2003 to a representative sample of 
household heads who live in these developments at that time. Hence, the survey data will 
provide a comparative before-after analysis from the housing development perspective. 50 



49 Komfeld and Bloom, 1999. 

S0 The original survey sampling plan had a longitudinal component that would have followed up baseline 
survey respondents who moved away. This component was dropped, however, because so many baseline re- 
spondents moved before Jobs-Plus began and thus before they could be meaningfully exposed to it. 
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PRELIMINARY ASSESSMENT OF THE IMPACT 
EVALUATION STRATEGY 



Although the impact evaluation is still in progress and final judgments about its 
methodology cannot yet be made, it is possible at this time to offer a brief preliminary 
assessment using data from the baseline survey and UI wage records. 

Evidence from the Baseline Survey 

Perhaps the single most important key to the success of our future Jobs-Plus im- 
pact estimates is the initial comparability of the Jobs-Plus and comparison samples. One 
important source of information to assess this comparability is the baseline survey con- 
ducted at all of the housing developments in our sample. Early tabulations of these find- 
ings are quite promising, indicating that the Jobs-Plus and comparison samples are simi- 
lar in many important ways. 51 

Table 4 illustrates this similarity for the overall pooled sample of sites. Given the 
focus of Jobs-Plus on promoting resident employment, the table lists comparisons in 
terms of employment-related baseline characteristics. 

The first column in the table reports the mean value of each characteristic for the 
six housing developments that were randomly assigned to the Jobs-Plus program (includ- 
ing Seattle). 52 Each value in the column is the mean of the corresponding means for the 
six Jobs-Plus developments. The second column in the table reports each characteristic 
for the pooled comparison group. Each value in this column is the mean of the corre- 
sponding means for the six local comparison groups. 53 

On average, it appears that the Jobs-Plus group and comparison group are quite 
comparable in terms of baseline characteristics related to their likely future labor market 
success. The percentage of sample members that were employed full time when the base- 
line survey was administered is identical for the two groups (43 percent); the percentage of 
the two groups whose household had received Food Stamps during the previous 12 months 
was almost identical (67 percent versus 66 percent); and the percentage whose household 
had received welfare during the past 12 months was very similar (51 percent to 49 percent). 
In terms of education level, a key factor related to future job market success, the two groups 
look quite similar (40 percent versus 42 percent had a high school diploma). 



51 Manpower Demonstration Research Corporation, 1999, Volumes I and II. 

52 Hence, only the randomly assigned Jobs-Plus development in Los Angeles was included. 

53 The comparison group mean for each site with two comparison developments is the simple mean of 
the reported means for each comparison development. 

-30- 




36 



Table 4 



Selected Mean Baseline Characteristics of Heads of Household 
for the Pooled Sample of Jobs-Plus and Comparison Developments 



Characteristic 


Jobs-Plus De- 
velopments 


Comparison 

Developments 


Currently employed full time 
(30+ hours per week) (%) 
Yes 


43 


43 


No 


57 


57 


Household received Food Stamps 
during past 12 months (%) 

Yes 


67 


66 


No 


32 


33 


Household received welfare 
during past 12 months (%) 
Yes 


51 


49 


No 


49 


51 


Educational attainment (%) 
GED certificate 


13 


15 


High school diploma 


40 


42 


Neither 


48 


43 



SOURCE: MDRC calculations from baseline survey data for each housing development that was randomly 
assigned at the six sites in the Jobs-Plus sample (including Seattle). 

NOTES: Distributions may not total 100 percent because of rounding. 



Evidence from UI Wage Records 

Baseline data from UI wage records are currently available for three Jobs-Plus 
sites: Baltimore, Dayton, and Los Angeles. Thus, it is possible to compare the baseline 
employment and earnings trends for their Jobs-Plus and comparison groups from the in- 
dividual perspective and the housing development perspective. In addition, it is possible 
to compare the baseline trends for their corresponding pooled Jobs-Plus and comparison 
groups. To keep the discussion brief, we focus only on employment trends. Correspond- 
ing results for earnings (not reported) yield virtually the same results. 

Baseline experience from the individual perspective 

As noted earlier, measuring Jobs-Plus impacts from the individual perspective 
addresses the question: “How did the program affect the future experiences of a specific 
group of people who were living in a program development at a particular time?” This 
requires observing the experience of the same persons over time, regardless of where 
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they live. Thus, to estimate impacts on employment and earnings from the individual per- 
spective requires: choosing a cohort of residents to track backward and forward in time; 
acquiring their quarterly UI wage records to do so; constructing their baseline and fol- 
low-up histories; measuring the follow-up deviation from their baseline trend; and com- 
paring this deviation for the Jobs-Plus and comparison samples. 

For our current analysis we chose a cohort of individuals who were: (1) recorded 
by their local housing authority as living in a Jobs-Plus or comparison development dur- 
ing October 1998, (2) not identified by housing authority records as being disabled, and 
(3) between 21 and 61 years of age in October 1998. 

This 1998 cohort was chosen because Jobs-Plus began program operations (in 
varying degrees) at each site during the middle to latter part of the year. Disabled persons 
were excluded from the analysis because their employment problems are often far more 
extreme than those of nondisabled persons and they are not included in the main target 
group for Jobs-Plus. Persons over 61 years of age were excluded because they would 
reach retirement age soon after the follow-up period for the analysis began. Lastly, per- 
sons under 2 1 years of age were excluded because they were teenagers during most of the 
Jobs-Plus baseline period and, thus, much of their employment history is not relevant to 
their future labor market success. 

Based on this definition, there were 349, 334, and 379 members of the 1998 cohort 
from the Baltimore, Dayton and Los Angeles Jobs-Plus developments, respectively. In ad- 
dition, there were 539, 582, and 453 cohort members from each of their comparison groups 
(the pooled sample for the two comparison developments for each site). 

The pattern of employment over time for the 1998 cohort is described in terms of 
its quarterly employment rates. These rates were computed using data from state UI wage 
records. Each sample member was considered employed during a quarter if his or her UI 
wage records indicated that he or she had received some earnings during that quarter. If 
no earnings were recorded for the quarter the sample member was considered not em- 
ployed. A quarterly employment rate was thus computed for a group as the percentage of 
its members with some Ul-reported earnings for the quarter. 

Figure 3 presents the quarterly employment histories of the 1998 Jobs-Plus cohort 
and comparison cohort from Baltimore, Dayton, and Los Angeles for a baseline period 
that begins in the first quarter of 1992 and ends in the latter part of 1998 (the exact final 
quarter to be included in the baseline period for our impact analysis will vary somewhat 
by site). 54 Figure 4 presents corresponding findings for the pooled groups from the three 
sites. Several striking results emerge from the figures. 

Employment rates rose dramatically for all of the groups during their baseline 
period. In Baltimore, these rates increased from about 20 percent to 40 percent; in Day- 
ton they increased from about 30 percent to 60 percent; and in Los Angeles they in- 
creased from about 30 percent to 45 percent. 



54 UI wage records were not available for Quarter 3 of 1993 in Dayton. 
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Figure 3 

Quarterly Percentage Employed During the Jobs-Plus Baseline Period 
for Nondisabled Adults, Ages 21-61, 
from the 1998 Cohort of the Program and Comparison Developments 
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Figure 4 

Quarterly Percentage Employed During the Jobs-Plus Baseline Period 
for Nondisabled Adults, Ages 21-61, 

from the 1998 Cohort of the Baltimore, Dayton, and Los Angeles Program 
and Comparison Developments (Pooled) 




One likely explanation for part of this shift is that many public housing resi- 
dents — just like many other Americans — probably responded positively to the many jobs 
created by the strong US economy. This challenges existing negative stereotypes about 
public housing residents being unwilling to work. 

Other pieces of the explanation may lie in the increased work incentives for per- 
sons with limited skills and experience produced by major changes in federal policy. Per- 
haps most important among these incentives is the Earned Income Tax Credit, which in- 
creased from a small income supplement program in the early 1990s to one providing 
over $30 billion to low-income workers in 2000 (Blank and Schmidt, 2001). 

Another potential explanatory factor is welfare reform, which began during the 
early 1990s with special state programs made possible by waivers of federal welfare 
regulations and culminated in the Personal Responsibility and Work Opportunity Recon- 
ciliation Act of 1996. As noted earlier, this new legislation replaced the existing federal 
welfare program, AFDC, with a radically different program, TANF, which contains many 
provisions designed to stimulate employment among welfare recipients. 

A further potential work incentive for persons with limited skills and job experi- 
ence was produced by legislated increases in the federal minimum wage rate from $4.25 
in 1992 to $4.75 in October 1996 to $5.15 in September 1997. 

The Jobs-Plus and comparison groups are matched very well. For each site, the 
baseline histories for the Jobs-Plus group and the comparison group are very similar. In 
fact, they practically “sit on top of each other.” In very few quarters is there a noticeable 
difference between their employment rates, and in even fewer quarters is the difference 
statistically significant (results of these significance tests are not shown). 
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This finding is consistent with previous results from Bloom and Glispie, 1999, for 
the Cleveland Jobs-Plus site, which as noted earlier, had participated in the initial phase 
of the demonstration. Therefore, in all four sites studied to date, the baseline employment 
histories of the Jobs-Plus and comparison groups are matched very well. 

Furthermore, the Jobs-Plus and comparison group match should be even tighter 
for analyses that pool findings across sites — the primary focus of our impact analysis. 
This is because pooled results benefit directly from the statistical properties of random 
assignment. Figure 4 presents such a pooled analysis of quarterly employment rates for 
the 1998 cohort from Baltimore, Dayton and Los Angeles. As can be seen, its baseline 
trend is even more stable than that for a single site, because of the larger sample for the 
pooled analysis. In addition, the baseline trends for the Jobs-Plus and comparison groups 
are even more similar to each other for the pooled analysis, because random differences 
for any single site tend to be offset by countervailing differences at other sites. 

The pronounced increase in employment rates experienced by public housing 
residents reduced the margin for Jobs-Plus to “ make a difference " in this outcome, al- 
though considerable room for improvement still remained when Jobs-Plus was launched. 
Because employment rates for public housing residents had been rising rapidly for some 
time before Jobs-Plus began, the margin for it to increase these rates had diminished ap- 
preciably. This situation was most pronounced in Dayton, where employment rates had 
reached 60 percent by the time Jobs-Plus program operations had gotten underway. It 
was less pronounced in Baltimore and Los Angeles. 

Baseline experience from the housing development perspective 

As noted earlier, measuring Jobs-Plus impacts from the housing development per- 
spective addresses the question: “How did the program affect conditions in its target de- 
velopments?” To answer this question requires comparing the experiences of persons 
who were living in the Jobs-Plus and comparison developments each quarter. Thus, to 
accomplish this task involves: determining who lived in each development each quarter, 
acquiring residents’ UI wage records, using these wage records to compute baseline and 
follow-up employment rates for the Jobs-Plus and comparison developments, and com- 
paring their follow-up deviations from their baseline trends. 

For each year, we define an annual resident cohort for a housing development as 
the persons who lived there during October. Thus, we start with knowledge of who was 
living in the development during the fourth calendar quarter. We did not, however, col- 
lect separate information about who lived in a development each quarter because of the 
high cost of doing so. Instead, we assumed that the residents known to be present during 
the fourth quarter of each year were also present during the immediately adjacent quar- 
ters. 



Thus, for example, we defined the first annual cohort to comprise residents in Oc- 
tober of 1992 (during quarter four) and assigned this same group to quarter three of 1992 
(the immediately preceding quarter) and quarter one of 1993 (the immediate succeeding 
quarter). We defined the next annual cohort as residents in October 1993, and assigned 
this group to its immediately adjacent quarters, and so on. 
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Given the high rate of mobility exhibited by public housing residents, it seemed 
too strong to assume that the same residents who were present during the fourth quarter 
were also present during the second quarter of each year. Therefore, we do not present 
findings for the second quarter of each year. Lastly, note that, for reasons discussed ear- 
lier, we only focus on nondisabled persons who were between 21 and 61 years old (in 
October). 

Figure 5 illustrates the baseline pattern of quarterly employment rates for the 
Jobs-Plus and comparison developments from each site, and Figure 6 illustrates the com- 
posite results for the pooled sites. To help distinguish this analysis from its counterpart 
for the individual perspective, we present it as a time-series of bar graphs instead of a line 
graph. The black bars in each figure represent quarterly employment rates for a Jobs-Plus 
development; the white bars represent corresponding findings for its pooled comparison 
developments. 

These findings indicate that employment conditions in each development changed 
over time in a way that was very similar to the results presented earlier for individual 
members of the 1998 cohort. Basically: (1) employment rates for all groups increased 
throughout the baseline period, (2) the baseline trends were virtually the same for the 
Jobs-Plus and comparison groups, especially for the pooled sites, and (3) employment 
rates were higher than expected when Jobs-Plus began program operations. 

Thus, from the housing development perspective it appears that: (1) employment 
conditions in public housing developments have improved substantially in response to the 
strong U.S. economy and recent major changes in federal policy, (2) there is a good match 
between the employment conditions of the Jobs-Plus and comparison developments, and 
(3) the margin for Jobs-Plus to make a difference diminished somewhat over the baseline 
period but still remained when the program was launched. 

It should also be noted that the baseline employment trends evident in the admin- 
istrative records data bode well for the survey analysis that will be conducted as part of 
the development-level impact study. As previously indicated, the survey data will permit 
only a “before-after” comparison of outcomes across developments, not a comparative 
interrupted time-series analysis, to estimate Jobs-Plus impacts on survey-based measures. 
However, the striking comparability of the pre-program employment trends across the 
program and comparison developments reinforces the conclusion that this simpler “be- 
fore-after” impact analysis is likely to be unbiased. 
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Figure 5 

Quarterly Percentage Employed During the Jobs-Plus Baseline Period 
for Nondisabled Adults, Ages 21-61, 

Who Were Residents (at the Time) of the 
Program and Comparison Developments 
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Figure 6 



Quarterly Percentage Employed During the Jobs-Plus Baseline Period 
for Nondisabled Adults, Ages 21-61, 

Who Were Residents (at the Time) 

of the Baltimore, Dayton, and Los Angeles Program and Comparison Developments (Pooled) 
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CONCLUDING THOUGHTS 

To date, Jobs-Plus program operations have been launched in six U.S. cities 
(counting Seattle) and its research design has been maintained in the field for almost six 
years. For all sites, a comprehensive baseline survey has been conducted; quarterly UI 
wage records are being collected; 53 extensive field research on program implementation 
has been conducted; and a number of project reports and papers have been produced (see 
Appendix A). Over the next two years, as the project approaches its conclusion, we will 
conduct a follow-up survey, obtain time-series data from state administrative records on 
the welfare receipt of sample members, finalize the collection of UI wage records on em- 
ployment and earnings for the project follow-up period, and complete our field research 
on program implementation. These efforts will result in a report on shorter-term impact 
findings (merged with implementation findings) in mid-2003 and a final project report 
with longer-term impact results in mid-2004. 

From our experiences with Jobs-Plus we have learned a number of important les- 
sons about using place-based random assignment to evaluate a comprehensive community 
initiative. In addition, our baseline findings have uncovered a puzzle that has intriguing 
methodological implications. We conclude this paper with a brief discussion of these issues. 



55 Due to state restrictions on the release of UI wage records for individuals in Tennessee it was not 
possible to obtain them for the Chattanooga Jobs-Plus site. After lengthy negotiations, however, it now 
appears that we will soon get this information in the form of averages for small groups of roughly 1 0 per- 
sons each. This information will make it possible to conduct almost all of our planned analyses from these 
data. 
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Lessons Learned About Place-Based Random Assignment 

From our experience using place-based random assignment to measure the im- 
pacts of Jobs-Plus we have learned that: 

• It was possible to put such a research design into effect. When Jobs-Plus 
was being planned, it was not at all clear whether random assignment of pub- 
lic housing developments would be possible. However, given the widely ac- 
knowledged importance of the policy problem being addressed, there was a 
strong desire by many potential sites to participate in the initiative, even with 
its research requirements. As noted earlier, 41 of the 50 cities contacted about 
Jobs-Plus stated an interest in participating, and all 1 5 of the cities chosen to 
submit a project application expressed a willingness to abide by its research 
requirements. Thus, it was possible to conduct place-based random assign- 
ment in the seven cities chosen to launch the initiative (including Cleveland, 
which left the project for reasons unrelated to random assignment). 

• It was necessary (and possible) to keep the research design in place for a 
number of years. Given the many difficulties and delays confronted when 
implementing Jobs-Plus (a common feature of all comprehensive community 
initiatives), it took a number of years for the program to materialize in full 
form. Thus, it was necessary to maintain the Jobs-Plus research design in the 
field for the past six years; and it will remain necessary to do so for another 
year in order to complete our evaluation. 

• A sustained and intensive effort was required to keep the research design in 
place and to maintain its integrity. Fending off threats (often successfully and 
sometimes not) to the existence or integrity of the Jobs-Plus program and its 
research design was a full-time, long-term job that required constant vigilance 
and rapid response. This, in turn, required the organizational capacity and re- 
sources to keep abreast of what was happening in the field; provide the tech- 
nical assistance necessary for sites to design, launch, and operate their pro- 
grams within the constraints of the Jobs-Plus research design; interact fre- 
quently and effectively with local decisionmakers; and contend in real time 
with a wide range of anticipated and unanticipated problems. 

• Where feasible, embedding a comparative interrupted time-series analysis 
within a place-based random assignment design can improve estimates of 
program impacts appreciably. These improvements can manifest themselves 
in at least three ways. First, adding an interrupted time-series component 
makes it possible to produce rigorous site-specific impact estimates, which is 
not possible with place-based random assignment alone. Second (and relat- 
edly), since the time-series component improves the impact estimate for each 
site, it also improves the pooled impact estimate for all sites. This feature is 
particularly important for evaluations with a small number of sites where the 
strength of random assignment is limited by the small number of units ran- 
domized. Third, the time-series component makes it possible to bridge the 
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methodological gap between people and place. This gap, which is created by 
residential mobility, is a major problem for all evaluations of place-based ini- 
tiatives . 56 However, by framing an interrupted time-series analysis from two 
different perspectives (for specific individuals over time and for specific 
places over time) one can obtain a rich understanding of program impacts. 



An Intriguing Puzzle 

In concluding our paper, it is impossible to ignore the striking, unanticipated, and 
currently inexplicable finding that, for each of the Jobs-Plus sites examined in the present 
paper (Baltimore, Dayton and Los Angeles) and for a former site examined previously 
(Cleveland ), 57 the baseline employment and earnings trends for the Jobs-Plus and com- 
parison groups are almost the same. This finding held both from the perspective of indi- 
vidual public housing residents and that of whole public housing developments. In addi- 
tion, it was consistent for a long period of time — seven years. Furthermore, it reflects a 
period of dramatic change in the U.S. economy and in federal policies with respect to 
low-income persons. For all of these reasons, the matches that we observe seem too good 
to be true. 

As a first step toward trying to explain these “matches made in heaven,” it is use- 
ful to consider factors that are not likely to have produced them. Of most relevance to the 
present discussion is that place-based random assignment is probably not responsible. 
This procedure had very little influence at the site level for Jobs-Plus because of the 
small numbers of units randomized at this level (two or three housing developments per 
site). In fact, given the high degree of similarity among developments at each site, it 
seems to make almost no difference which ones were chosen for Jobs-Plus or the com- 
parison group and whether they were chosen with or without random assignment. 

A second factor that probably was not responsible for the close matches we ob- 
served was the process used to construct them. This process was very simple given the 
limited information available and the small numbers of eligible housing developments to 
work with at each site. Thus, sophisticated matching procedures were neither warranted 
nor feasible. All that we did to choose a pool of eligible housing developments (as de- 
scribed earlier) was to use simple thresholds based on their size, employment rates and 
welfare receipt rates plus PHA judgments about their likely suitability for conducting a 
Jobs-Plus program. And all that we did to choose among each site’s pool of eligible de- 
velopments to construct a triplet or pair for random assignment was to apply limited 
judgments about the comparability of their residents’ background characteristics. 

While at this point it is unclear what produced the striking matches that we ob- 
served, we offer a couple of hypotheses for consideration. First, our findings may have 
occurred simply by chance (they merely represent good luck). If this is the case, then we 



56 Mobility also creates major problems for the programs themselves, since it limits the potential expo- 
sure of their target populations. 

57 Bloom and Glispie, 1 999. 
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may begin to see different results when corresponding data for the remaining three Jobs- 
Plus sites — St. Paul, Chattanooga and Seattle — become available. 

A second hypothesis is that our study population — able-bodied working-age adults 
who live in public housing — is especially homogenous within a given city. Although the 
labor market success of public housing residents may vary widely across cities (reflecting 
differences in their local economic conditions and PHA tenant selection policies), they may 
vary far less across developments within cities. If this is the case, we may continue to see 
close matches between the Jobs-Plus and comparison groups when data for our three re- 
maining sites become available. 

In closing, we would like to briefly compare the preceding Jobs-Plus implications 
for geographic matching (with or without place-based random assignment) with those 
from two MDRC methodological studies — one just completed and another just getting 
underway. The first study focused on welfare-to-work programs. It sought to determine 
(among many other things) how well welfare recipients from one set of local welfare of- 
fices in a city serve as a nonexperimental comparison group for recipients from other of- 
fices in the same city . 58 Using a wide range of statistical and econometric matching and 
modeling procedures, it was found that welfare recipients from one part of a city did not 
usually make a good comparison group for those from another part of the city. Further- 
more, and more troubling, it was impossible to predict whether or when such a compari- 
son group would or would not work. All of these findings were based on the use of ran- 
dom assignment experiments in all sites, which made it possible to judge how well each 
nonexperimental comparison group was working relative to what was known from the 
experimental evidence. 

Findings from this study directly conflict with those from Jobs-Plus, even though 
the relevant populations and institutional situations from both studies share important 
common features. First of all, both study populations are defined narrowly in terms of 
receiving a particular form of public assistance (public housing for Jobs-Plus and welfare 
payments for welfare-to-work programs). Hence, they are arguably more homogeneous 
than a general population of low-income persons. Secondly, comparison groups for both 
studies were chosen on a geographic basis within a given city (through public housing 
developments for Jobs-Plus and through local welfare offices for welfare-to-work pro- 
grams). These two similarities make it doubly difficult to understand why the two studies 
yield such divergent implications for geographically based comparison groups. 

A second, more limited point of reference is a small-scale pilot study being con- 
ducted by MDRC to assess the extent to which low-income neighborhoods (of three to 
five census tracts each) that are matched on poverty-related factors at one point in time 
remain well matched subsequently . 59 This research is being conducted as part of our plan- 
ning for a next generation of community studies. Its goal is to assess empirically the 
likely internal validity of nonexperimental matched neighborhood designs for measuring 



58 Bloom, Michalopoulos, Hill and Lei, 2002. This study also examined the effectiveness of compari- 
son groups constructed in many ways other than on the basis of local welfare offices in the same city. 

59 Verma, 2002b. This analysis is based on data obtained for MDRC’s four-city comprehensive Project 
on Devolution and Urban Change. 
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labor market impacts. Preliminary unpublished findings based on data for Cleveland sug- 
gest that at least in one city for one outcome using one neighborhood matching procedure 
for one time period, neighborhoods that are similar at baseline remain similar for at least 
four years thereafter. These findings are consistent with those for Jobs-Plus. 

In conclusion, based on all of the information that we could readily bring to bear, 
the striking baseline comparability observed for the Jobs-Plus program groups and the 
comparison groups represents an intriguing puzzle that has supporting and conflicting re- 
search precedents and important implications for future evaluations of place-based initia- 
tives. 
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Program Implementation Studies 

• Mobilizing Public Housing Communities for Work : Origins and Early Accomplishments 
of the Jobs-Plus Demonstration. James A. Riccio. 1999. 

• Jobs-Plus Site-by-Site: An Early Look at Program Implementation. Edited by Susan Phil- 
ipson Bloom with Susan Blank. 2000. 

• Building New Partnerships for Employment. Linda Y. Kato and James A. Riccio, with 
Jennifer Dodge. 2001. 

• Making Work Pay for Public Housing Residents: Financial-Incentive Designs at Six 
Jobs-plus Demonstration Sites. Cynthia Miller and James A. Riccio. 2002. 

• The Special Challenges of Offering Employment Programs in Culturally Diverse Com- 
munities: The Jobs-Plus Experience in Public Housing Developments. Linda Y. Kato. 
2002. 

• The Employment Experiences of Public Housing Residents: Findings from the Jobs-Plus 
Baseline Survey. John Martinez. 2002. 
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• Welfare , Housing , and Employment. Susan Blank and James A. Riccio. 2001. 

• Promoting Employment in Public Housing Communities. Steven Bliss and 
James A. Riccio. 2001. 

« Making Work Pay for Public Housing Residents. James A. Riccio and Steven Bliss. 2002. 

Baseline Survey Data Books 
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Other Publications Related to Jobs-Plus 

• Are Public Housing Residents Really Harder to Employ? 2001 . James A. Riccio and 
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• “Lessons from Welfare-to-Work Experiments and Their Implications for Housing-Based 
Self-Sufficiency Programs.” 1999. James A. Riccio. In The Home Front: Implications of 
Welfare Reform for Housing (Sandra Newman, ed.). Washington, D.C.: Urban Institute 
Press. 
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IN PROGRESS 

The Circumstances of Children Living in Public Housing Prior to the Start of Jobs-Plus 
(late 2002). 

Staying or Leaving: A Study of Residents ' Decisions to Remain in or Move Out of Jobs- 
Plus Public Housing Developments (late 2002). 

Jobs-Plus Site-by-Site: Key Features of Six Mature Jobs-Plus Programs (late 2002). 
Residents ' patterns of participation in Jobs-Plus (early 2003). 

Implementation and use of financial incentives (early 2003). 

PLANNED 

Implementation of community support for work (mid 2003). 

Interim impact and implementation report (mid 2003). 

Final impact and implementation report: (mid 2004). 
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